Motion Vector Dependent Spatial Transformation in Video Coding

ABSTRACT

Coding efficiency may be improved by subdividing a block into smaller sub-blocks for prediction. A first rate distortion value of a block optionally partitioned into smaller prediction sub-blocks of a first size is calculated using respective inter prediction modes and transforms of the first size. The residuals are used to encode the block using a transform of a second size smaller than the first size, generating a second rate distortion value. The values are compared to determine whether coding efficiency gains may result from inter predicting the smaller, second size sub-blocks. If so, the block is encoded by generating prediction residuals for the second size sub-blocks, and neighboring sub-blocks are grouped, where possible, based on common motion information. Each resulting composite residual block is transformed by a transform of the same size to generate another rate distortion value. The encoded block with the lowest rate distortion value is used.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.13/764,020, filed Feb. 11, 2013, the entire content of which isincorporated herein by reference.

BACKGROUND

Digital video streams typically represent video using a sequence offrames or still images. Each frame can include a number of blocks, whichin turn may contain information describing the value of color,brightness or other attributes for pixels. The amount of data in atypical video stream is large, and transmission and storage of video canuse significant computing or communications resources. Variousapproaches have been proposed to reduce the amount of data in videostreams, including compression and other encoding techniques.

SUMMARY

Disclosed herein are aspects of systems, methods and apparatuses forcoding a video stream having a plurality of frames. One method describedherein includes encoding a block a first time by encoding at least onesub-block of the block having a first size using a respective interprediction mode and a transform having the first size, encoding theblock a second time, after encoding the block the first time, byencoding the at least one sub-block of the block having the first sizeusing the respective inter prediction mode and a transform having asecond size smaller than the first size, performing inter prediction ofthe block by performing inter prediction of a plurality of sub-blocks ofthe block having the second size to generate a respective plurality ofsub-block residuals, combining one or more of the plurality of sub-blockresiduals to form at least one composite block using motion informationassociated with respective ones of the plurality of sub-blocks, encodingthe block a third time, after encoding the block the second time, bytransforming sub-block residuals of each composite block of the at leastone composite block using a respective transform corresponding to a sizeof each composite block, and selecting one of the encoded blocks as anoutput encoded block.

Another method described herein includes encoding a block a first timeto generate a first rate distortion value by encoding at least onesub-block of the block having a first size using a respective interprediction mode and a transform having the first size, encoding theblock a second time, after encoding the block the first time, togenerate a second rate distortion value by encoding the at least onesub-block of the block having the first size using the respective interprediction mode and a transform having a second size smaller than thefirst size, and responsive to the second rate distortion value beingless than the first rate distortion value, performing inter predictionof the block by performing inter prediction of a plurality of sub-blocksof the block having the second size to generate a respective pluralityof sub-block residuals, combining one or more of the plurality ofsub-block residuals to form at least one composite block using motioninformation associated with respective ones of the plurality ofsub-blocks, and encoding the block a third time, after encoding theblock the second time, to generate a third rate distortion value bytransforming sub-block residuals of each composite block of the at leastone composite block using a respective transform corresponding to a sizeof each composite block. The method also includes selecting an outputencoded block for the block based on the rate distortion values.

An apparatus for encoding a video stream having a plurality of frames,according to the teachings herein, includes a memory and a processor. Inan implementation, the processor is configured to execute instructionsstored in the memory to encode a block a first time by encoding at leastone sub-block of the block having a first size using a respective interprediction mode and a transform having the first size, encode the blocka second time, after encoding the block the first time, by encoding theat least one sub-block of the block having the first size using therespective inter prediction mode and a transform having a second sizesmaller than the first size. perform inter prediction of the block byperforming inter prediction of a plurality of sub-blocks of the blockhaving the second size to generate a respective plurality of sub-blockresiduals, combine one or more of the plurality of sub-block residualsto form at least one composite block using motion information associatedwith respective ones of the plurality of sub-blocks, encode the block athird time, after encoding the block the second time, by transformingsub-block residuals of each composite block of the at least onecomposite block using a respective transform corresponding to a size ofeach composite block, and select one of the encoded blocks as an outputencoded block.

Variations in these and other aspects will be described in additionaldetail hereafter.

BRIEF DESCRIPTION OF THE DRAWINGS

The description herein makes reference to the accompanying drawingswherein like reference numerals refer to like parts throughout theseveral views.

FIG. 1 is a schematic diagram of a video encoding and decoding system.

FIG. 2 is a block diagram of an exemplary computing device that canimplement a transmitting station or a receiving station.

FIG. 3 is a diagram of an example video stream to be encoded andsubsequently decoded.

FIG. 4 is a block diagram of a video encoder in accordance with animplementation of the teachings herein.

FIG. 5 is a block diagram of a video decoder in accordance with animplementation of the teachings herein.

FIG. 6 is a flowchart diagram of a process for encoding a video streamaccording to an aspect of the teachings herein.

FIG. 7 is a flowchart diagram of a process for decoding a video streamaccording to an aspect of the teachings herein.

FIG. 8 is a diagram of a block and its sub-blocks according to an aspectof the teachings herein.

DETAILED DESCRIPTION

Real-time video streaming, multi-point video conferencing or videobroadcasting are examples of applications that employ video streamencoding including compression. One compression technique includesperforming inter prediction on the video stream, wherein blocks of aframe of a video stream are predicted using blocks of data from otherframes (e.g., temporally nearby frames) or from other portions of theframe. To encode the block, the encoder tests various partitions of theblock (such as splitting a block comprising 16×16 pixels into 8×8 pixelpartitions, 4×4 pixel partitions or a combination thereof), performs amotion search per partition, and then forms a prediction block for eachpartition. The differences between pixel values of the respectiveprediction blocks (the “residuals”) are then coded, representing areduction in data over the original block. The partitions are sometimesreferred to as sub-blocks. After generation of the residuals,transform-based encoders may apply a transform having a first size tocertain of the residuals, such as applying a 4×4 transform when thepartition dimension is 8×4, 4×8 or 4×4, while applying a transformhaving a second size to others of the residuals, such as applying an 8×8transform when the partition dimension is 16×8, 8×8 or 8×16. Unlessotherwise noted, when used herein, size refers to a set of dimensions.

In such an arrangement, the grid used for the transforms is fixed. Incontrast, according to the teachings herein, the dimension of thetransform used for motion compensated prediction residuals is selecteddepending on the motion information, including partition, motion vectorand reference frame(s). In this way, a low complexity approach tosearching for the coding mode decisions that minimize rate distortioncost is developed.

First discussed below are environments in which aspects of thisdisclosure can be implemented, and then details of certainimplementations are explained.

FIG. 1 is a schematic diagram of a video encoding and decoding system100. An exemplary transmitting station 112 can be, for example, acomputer having an internal configuration of hardware such as thatdescribed in FIG. 2. However, other suitable implementations oftransmitting station 112 are possible. For example, the processing oftransmitting station 112 can be distributed among multiple devices.

A network 128 can connect transmitting station 112 and a receivingstation 130 for encoding and decoding of the video stream. Specifically,the video stream can be encoded in transmitting station 112 and theencoded video stream can be decoded in receiving station 130. Network128 can be, for example, the Internet. Network 128 can also be a localarea network (LAN), wide area network (WAN), virtual private network(VPN), cellular telephone network or any other means of transferring thevideo stream from transmitting station 112 to, in this example,receiving station 130.

Receiving station 130, in one example, can be a computer having aninternal configuration of hardware such as that described in FIG. 2.However, other suitable implementations of receiving station 130 arepossible. For example, the processing of receiving station 130 can bedistributed among multiple devices.

Other implementations of video encoding and decoding system 100 arepossible. For example, an implementation can omit network 128. Inanother implementation, a video stream can be encoded and then storedfor transmission at a later time to receiving station 130 or any otherdevice having memory. In one implementation, receiving station 130receives (e.g., via network 128, a computer bus, and/or somecommunication pathway) the encoded video stream and stores the videostream for later decoding. In an exemplary implementation, a real-timetransport protocol (RTP) is used for transmission of the encoded videoover network 128. In another implementation, a transport protocol otherthan RTP may be used, e.g., a Hyper Text Transfer Protocol (HTTP)-basedvideo streaming protocol.

FIG. 2 is a block diagram of an exemplary computing device 200 that canimplement a transmitting station or a receiving station. For example,computing device 200 can implement one or both of transmitting station112 and receiving station 130 of FIG. 1. Computing device 200 can be inthe form of a computing system including multiple computing devices, orin the form of a single computing device, for example, a mobile phone, atablet computer, a laptop computer, a notebook computer, a desktopcomputer, and the like.

A CPU 224 in computing device 200 can be a conventional centralprocessing unit. Alternatively, CPU 224 can be any other type of device,or multiple devices, capable of manipulating or processing informationnow-existing or hereafter developed. Although the disclosedimplementations can be practiced with a single processor as shown, e.g.,CPU 224, advantages in speed and efficiency can be achieved using morethan one processor.

A memory 226 in computing device 200 can be a read only memory (ROM)device or a random access memory (RAM) device in an implementation. Anyother suitable type of storage device can be used as memory 226. Memory226 can include code and data 227 that is accessed by CPU 224 using abus 230. Memory 226 can further include an operating system 232 andapplication programs 234, the application programs 234 including atleast one program that permits CPU 224 to perform the methods describedhere. For example, application programs 234 can include applications 1through N, which further include a video coding application thatperforms the methods described here. Computing device 200 can alsoinclude a secondary storage 236, which can, for example, be a memorycard used with a mobile computing device 200. Because the videocommunication sessions may contain a significant amount of information,they can be stored in whole or in part in secondary storage 236 andloaded into memory 226 as needed for processing.

Computing device 200 can also include one or more output devices, suchas a display 228. Display 228 may be, in one example, a touch sensitivedisplay that combines a display with a touch sensitive element that isoperable to sense touch inputs. Display 228 can be coupled to CPU 224via bus 230. Other output devices that permit a user to program orotherwise use computing device 200 can be provided in addition to or asan alternative to display 228. When the output device is or includes adisplay, the display can be implemented in various ways, including by aliquid crystal display (LCD), a cathode-ray tube (CRT) display or lightemitting diode (LED) display, such as an organic LED (OLED) display.

Computing device 200 can also include or be in communication with animage-sensing device 238, for example a camera, or any otherimage-sensing device 238 now existing or hereafter developed that cansense an image such as the image of a user operating computing device200. Image-sensing device 238 can be positioned such that it is directedtoward the user operating computing device 200. In an example, theposition and optical axis of image-sensing device 238 can be configuredsuch that the field of vision includes an area that is directly adjacentto display 228 and from which display 228 is visible.

Computing device 200 can also include or be in communication with asound-sensing device 240, for example a microphone, or any othersound-sensing device now existing or hereafter developed that can sensesounds near computing device 200. Sound-sensing device 240 can bepositioned such that it is directed toward the user operating computingdevice 200 and can be configured to receive sounds, for example, speechor other utterances, made by the user while the user operates computingdevice 200.

Although FIG. 2 depicts CPU 224 and memory 226 of computing device 200as being integrated into a single unit, other configurations can beutilized. The operations of CPU 224 can be distributed across multiplemachines (each machine having one or more of processors) that can becoupled directly or across a local area or other network. Memory 226 canbe distributed across multiple machines such as a network-based memoryor memory in multiple machines performing the operations of computingdevice 200. Although depicted here as a single bus, bus 230 of computingdevice 200 can be composed of multiple buses. Further, secondary storage236 can be directly coupled to the other components of computing device200 or can be accessed via a network and can comprise a singleintegrated unit such as a memory card or multiple units such as multiplememory cards. Computing device 200 can thus be implemented in a widevariety of configurations.

FIG. 3 is a diagram of an example of a video stream 350 to be encodedand subsequently decoded. Video stream 350 includes a video sequence352. At the next level, video sequence 352 includes a number of adjacentframes 354. While three frames are depicted as adjacent frames 354,video sequence 352 can include any number of adjacent frames. Adjacentframes 354 can then be further subdivided into individual frames, e.g.,a single frame 356. At the next level, a single frame 356 can be dividedinto a series of planes or segments 358. Segments 358 can be subsets offrames that permit parallel processing, for example.

Segments 358 include blocks 360, which can contain data correspondingto, for example, 16×16 pixels in frame 356. Blocks 360 can also be ofany other suitable size such as 4×4, 8×8 16×8, 8×16, 16×16 or larger.Unless otherwise noted, the terms block and macroblock are usedinterchangeably herein.

FIG. 4 is a block diagram of an encoder 470 in accordance with animplementation. Encoder 470 can be implemented, as described above, intransmitting station 112 such as by providing a computer softwareprogram stored in memory, for example, memory 226. The computer softwareprogram can include machine instructions that, when executed by aprocessor such as CPU 224, cause transmitting station 112 to encodevideo data in the manner described in FIG. 4. Encoder 470 can also beimplemented as specialized hardware included in, for example,transmitting station 112. Encoder 470 has the following stages toperform the various functions in a forward path (shown by the solidconnection lines) to produce an encoded or compressed bitstream 488using video stream 350 as input: an intra/inter prediction stage 472, atransform stage 474, a quantization stage 476, and an entropy encodingstage 478. Encoder 470 may also include a reconstruction path (shown bythe dotted connection lines) to reconstruct a frame for encoding offuture blocks. In FIG. 3, encoder 470 has the following stages toperform the various functions in the reconstruction path: adequantization stage 480, an inverse transform stage 482, areconstruction stage 484, and a loop filtering stage 486. Otherstructural variations of encoder 470 can be used to encode video stream350.

When video stream 350 is presented for encoding, the frame 356 withinvideo stream 350 can be processed in units of blocks. At intra/interprediction stage 472, each block can be encoded using intra-frameprediction (also called intra prediction) or inter-frame prediction(also called inter prediction). In either case, a prediction block canbe formed for a block to be encoded. In the case of intra prediction, aprediction block may be formed from nearby data in the current framethat has been previously encoded and reconstructed. In the case of interprediction, a prediction block may be formed from data in one or morepreviously constructed reference frames identified through a motionsearch and associated with a motion vector.

Next, still referring to FIG. 4, the prediction block can be subtractedfrom the current block at intra/inter prediction stage 472 to produce aresidual block (also called a residual). Note that the block may besubdivided into smaller blocks for prediction and the calculation ofrespective residuals. Transform stage 474 transforms a residual intotransform coefficients in, for example, the frequency domain. Examplesof block-based transforms include the Karhunen-Loève Transform (KLT),the Discrete Cosine Transform (DCT), Asymmetrical Discrete SineTransform (ADST) and the Singular Value Decomposition Transform (SVD).In one example, the DCT transforms the block into the frequency domain.In the case of DCT, the transform coefficient values are based onspatial frequency, with the lowest frequency (DC) coefficient at thetop-left of the matrix and the highest frequency coefficient at thebottom-right of the matrix.

Quantization stage 476 converts the transform coefficients into discretequantum values, which are referred to as quantized transformcoefficients, using a quantizer value or a quantization level. Thequantized transform coefficients are then entropy encoded by entropyencoding stage 478. The entropy-encoded coefficients, together withother information used to decode the block, which may include forexample the type of prediction used, motion vectors and quantizer value,are then output to the compressed bitstream 488. Compressed bitstream488 can be formatted using various techniques, such as variable lengthcoding (VLC) or arithmetic coding. Compressed bitstream 488 can also bereferred to as an encoded video stream and the terms will be usedinterchangeably herein.

The reconstruction path in FIG. 4 (shown by the dotted connection lines)can be used to ensure that both encoder 470 and a decoder 500 (describedbelow) use the same reference frames to decode compressed bitstream 488.The reconstruction path performs functions that are similar to functionsthat take place during the decoding process that are discussed in moredetail below, including dequantizing the quantized transformcoefficients at dequantization stage 480 and inverse transforming thedequantized transform coefficients at inverse transform stage 482 toproduce a derivative residual block (also called a derivative residual).At reconstruction stage 484, the prediction block that was predicted atthe intra/inter prediction stage 472 can be added to the derivativeresidual to create a reconstructed block. Loop filtering stage 486 canbe applied to the reconstructed block to reduce distortion such asblocking artifacts.

Other variations of encoder 470 can be used to encode compressedbitstream 488. For example, a non-transform based encoder 470 canquantize the residual signal directly without transform stage 474. Inanother implementation, an encoder 470 can have quantization stage 476and dequantization stage 480 combined into a single stage.

FIG. 5 is a block diagram of a decoder 500 in accordance with anotherimplementation. Decoder 500 can be implemented in receiving station 130,for example, by providing a computer software program stored in memory226. The computer software program can include machine instructionsthat, when executed by a processor such as CPU 224, cause receivingstation 130 to decode video data in the manner described in FIG. 5.Decoder 500 can also be implemented in hardware included in, forexample, transmitting station 112 or receiving station 130.

Decoder 500, similar to the reconstruction path of encoder 470 discussedabove, includes in one example the following stages to perform variousfunctions to produce an output video stream 516 from compressedbitstream 488: an entropy decoding stage 502, a dequantization stage504, an inverse transform stage 506, an intra/inter prediction stage508, a reconstruction stage 510, a loop filtering stage 512 and adeblocking filtering stage 514. Other structural variations of decoder500 can be used to decode compressed bitstream 488.

When compressed bitstream 488 is presented for decoding, the dataelements within compressed bitstream 488 can be decoded by entropydecoding stage 502 (using, for example, arithmetic coding) to produce aset of quantized transform coefficients. Dequantization stage 504dequantizes the quantized transform coefficients, and inverse transformstage 506 inverse transforms the dequantized transform coefficients toproduce a derivative residual that can be identical to that created byinverse transform stage 482 in encoder 470. Using header informationdecoded from compressed bitstream 488, decoder 500 can use intra/interprediction stage 508 to create the same prediction block as was createdin encoder 470, e.g., at intra/inter prediction stage 472. Atreconstruction stage 510, the prediction block can be added to thederivative residual to create a reconstructed block. Loop filteringstage 512 can be applied to the reconstructed block to reduce blockingartifacts. Other filtering can be applied to the reconstructed block.For example, deblocking filtering stage 514 can be applied to thereconstructed block to reduce blocking distortion, and the result isoutput as output video stream 516. Output video stream 516 can also bereferred to as a decoded video stream and the terms will be usedinterchangeably herein.

Other variations of decoder 500 can be used to decode compressedbitstream 488. For example, decoder 500 can produce output video stream516 without deblocking filtering stage 514.

As mentioned briefly above, the dimension of the transform used formotion compensated prediction residuals may be selected depending onmotion information, including partition, motion vector and referenceframe(s) using a low complexity approach to searching for coding modedecisions. Details of certain implementations of a motion vectordependent spatial transformation are described below starting with FIG.6.

FIG. 6 is a flowchart of a process 600 for encoding a video streamaccording to an aspect of the teachings herein. Process 600 can beimplemented in a system such as encoder 470 to encode a video streamusing inter prediction and variably-sized transforms. Process 600 can beimplemented, for example, as a software program that is executed by acomputing device such as transmitting station 112 or receiving station130. The software program can include machine-readable instructions thatare stored in a memory such as memory 226 that, when executed by aprocessor such as CPU 224, cause the computing device to perform process600. Process 600 can also be implemented using hardware in whole or inpart. As explained above, some computing devices may have multiplememories and multiple processors, and the steps of process 600 may insuch cases be distributed using different processors and memories. Useof the terms “processor” and “memory” in the singular encompassescomputing devices that have only one processor or one memory as well asdevices having multiple processors or memories that may each be used inthe performance of some but not necessarily all of the recited steps.

For simplicity of explanation, process 600 is depicted and described asa series of steps. However, steps in accordance with this disclosure canoccur in various orders and/or concurrently. Additionally, steps inaccordance with this disclosure may occur with other steps not presentedand described herein. Furthermore, not all illustrated steps may berequired to implement a method in accordance with the disclosed subjectmatter.

At step 602, a motion search is performed to find motion vectors formotion compensated (i.e., inter) prediction for partitions or sub-blocksof a 4N×4N block, where the partitions are of size 2N×2N. In thisexample, a block may be of a size 16×16, so the sub-blocks comprise 8×8pixels. Inter prediction is a technique for reducing the number of bitsrequired to represent a block by subtracting video information from oneor more reference frames from the current block. Inter prediction canspatially translate the video data from a reference frame to moreclosely match the current block to compensate for motion of data betweenframes of video data. Motion vectors indicate how the reference data isto be translated for comparison with the current block or sub-block.

Various motion vectors are tested by encoding the block in step 604.Specifically, each motion vector is used to generate a prediction blockby translating pixels from the reference frame using the motion vector.Then, a respective residual block is generated by subtracting aprediction block from a sub-block. A transform having a dimension of2N×2N is applied to each resultant 2N×2N residual block. Finally, thetransformed blocks are quantized and entropy coded. Thus, the block isencoded by encoding its sub-blocks. The motion information thatminimizes the rate distortion cost is stored, along with the minimumrate distortion cost in step 606.

As can be recognized from this description, steps 602 and 604 correspondto a rate distortion loop. The rate, or number of bits, required toencode the block including additional bits added to the encoded videobitstream to indicate prediction modes, transform sizes and/or sub-blocksizes is measured along with the distortion for each combination ofprediction mode, transform sizes and/or sub-block sizes. Distortion canbe measured by encoding and decoding a block, then subtracting thedecoded block from the original un-encoded block. The difference is ameasure of the distortion introduced by the encoding. In thisimplementation, the rate distortion loop identifies as motioninformation the combination of motion vectors for the 2N×2N blocks thatresults in the lowest distortion for a given bit rate, for example,based on testing more than one 8×8 inter prediction mode.

At step 608, the motion information stored in step 606 is used togenerate residuals for the 2N×2N sub-blocks. In step 610, process 600then encodes the block using the residuals. Specifically, an N×Ntransform is applied to the residuals, followed by quantization andentropy coding. An N×N transform is applied to the 2N×2N residuals whereN=4, for example, such that four 4×4 transforms are applied to each offour 4×4 portions of each 8×8 residual. The rate distortion cost of thisencoding is calculated in a similar manner as described above.

At step 612, the rate distortion cost for encoding the 2N×2N sub-blocksusing 2N×2N transforms is compared with the rate distortion cost forencoding the 2N×2N sub-blocks using N×N transforms. Using N×N predictionmodes and N×N transforms to encode a block can improve the ratedistortion cost over using 2N×2N prediction modes and 2N×2N transforms.However, testing N×N prediction modes and transforms is very timeconsuming. Accordingly, an implementation uses the comparison of 2N×2Nprediction and 2N×2N transforms to 2N×2N prediction and N×N as a roughproxy to determine whether or not to search N×N partitions of thecurrent block. If performing N×N transforms on 2N×2N predicted data doesnot yield any improvements in rate distortion, then no N×N encodingmodes are tested.

More specifically, if the rate distortion cost resulting from using the2N×2N transforms is less than or equal to the rate distortion costresulting from using the N×N transforms at step 612, process 600branches to step 614, where a motion search is separately performed forboth 2N×4N sub-blocks and 4N×2N sub-blocks. At next step 616, eachmotion vector resulting from the motion search is used to generate aprediction block by translating pixels from the reference frame usingthe motion vector. Then, a respective residual block is generated bysubtracting a prediction block from a sub-block. A transform having adimension of 2N×4N is applied to each resultant 2N×4N residual block,and a transform having a dimension of 4N×2N is applied to each resultant4N×2N residual block. Finally, the transformed blocks are quantized andentropy coded. Thus, the block is encoded twice, once by encoding its2N×4N sub-blocks, and once by encoding its 4N×2N sub-blocks. The encodedblock with a minimal rate distortion cost for each of the 4N×2N and2N×4N sub-block partitions is stored, along with the minimum ratedistortion cost. This represents another rate distortion loop. Afterthis step, process 600 advances to step 624 discussed below.

In contrast, if the rate distortion cost resulting from using the 2N×2Ntransforms is greater than the rate distortion cost resulting from usingthe N×N transforms at step 612, process 600 branches to step 618, wheremotion-compensated prediction is performed for N×N sub-blocks of theblock. This step may involve performing a rate distortion loop thattests various N×N inter prediction modes for each sub-block, generatesresidual blocks for each mode and each sub-block, transforms theresidual blocks using an N×N transform, quantizes the transformedresidual blocks and entropy codes the quantizer coefficients.

The motion vector and reference frame that results in the smallest ratedistortion cost for each N×N sub-block is compared to its neighboringblocks and, if appropriate, the N×N residuals are combined in step 620.In this example, neighboring blocks that have substantially the samemotion information are aggregated to form a composite block. Forexample, if the reference frame is the same and the motion vector is thesame within a defined range, such as a ±10% difference in vector valuesor within a standard deviation of each other, etc., neighboring blocksmay be combined in a composite block. The combined residuals form acomposite block that is encoded in step 622. Specifically, eachcomposite block is transformed using a spatial transform having the samedimension as the composite block. Then, the transformed composite blockis quantized and then entropy encoded to calculate the rate distortionassociated with the N×N partition mode. Merging sub-blocks based onmotion information can improve encoding efficiency while preserving lowdistortion values.

Sub-blocks can be combined to create 2N×N or N×2N sub-blocks, 4N×N orN×4N sub-blocks, 2N×2N sub-blocks, 4N×2N or 2N×4N sub-blocks or a 4N×4Nsub-block, for example. Included in the possible sub-blocks of a blockare the cases where no sub-blocks can be merged, and the block includesall N×N sub-blocks, and the case where all of the sub-blocks are mergedtogether and the block includes one 4N×4N sub-block. FIG. 8 is a diagramof a block and its sub-blocks according to aspects of the teachingsherein. FIG. 8 shows a 16×16 block 800 with a 4×4 sub-block 802, an 8×8sub-block 804 and a 4×16 sub-block 806. The remaining sub-blocks ofblock 800 would also be of size 4×4 (that is, not combined with othersub-blocks), but are not shown for illustrative purposes.

Returning to FIG. 6, the rate distortion costs for encoding the blockwith encoding parameters including the various sub-blocks, interprediction modes and transformation modes are compared at step 622 andthe partition mode (i.e., sub-block size) that minimizes the ratedistortion cost is identified as the final encoding parameter. Forexample, if the 4N×2N and 2N×4N partition modes are tested in steps 614and 616, then the cost associated with each of these modes is comparedto the cost of the 2N×2N partition mode stored in step 606. Desirably,and although not shown in FIG. 6, a 4N×4N partition mode is also tested,that is, a mode where the block is not subdivided for prediction, togenerate a cost for this comparison once encoded. Thus, in step 624, oneof the partition modes 4N×4N, 2N×2N, 4N×2N or 2N×4N is selected based onthe mode resulting in the lowest cost. Then, in step 626, the encodedblock included in the video bitstream is the block encoded in step 604,one of the blocks encoded in step 616 or the encoded 4N×4N block.

In contrast, if the N×N partition mode is tested in steps 618, 620 and622, then the cost associated with the encoded composite block(s) instep 622 is compared to the cost of the 2N×2N partition mode stored instep 606. Similar to the description above, it is desirable that a 4N×4Npartition mode is also tested to generate an encoded 4N×4N block and itsassociated cost. Thus, in step 624, one of the partition modes 4N×4N,2N×2N or N×N is selected based on the mode resulting in the lowest cost.Then, in step 626, the encoded block included in the video bitstream isthe block encoded in step 604, the block encoded in step 622 (e.g., thecomposite block(s) generated using the N×N sub-blocks) or the encoded4N×4N block.

Bits that indicate the encoding parameters used to encode the block,such as the sub-block size (e.g., the prediction partition mode), motionvectors and, where applicable, the size and location of composite blocksare also included in the video bitstream, such as in a block header ofthe block. At step 628, process 600 checks to see if there are anyblocks of the frame to be processed. Process 600 analyzes blocks, forexample, in raster or other scan order. If all blocks are not done,process 600 loops back to step 602 to begin processing the next block.If all blocks are done, process 600 exits.

FIG. 7 is a flowchart diagram of a process 700 for decoding a videostream according to an aspect of the teachings herein. Process 700 canbe implemented in a system such as decoder 500 to decode a videobitstream generated using motion vector dependent spatialtransformation. Process 700 can be implemented, for example, as asoftware program that is executed by a computing device such astransmitting station 112 or receiving station 130. The software programcan include machine-readable instructions that are stored in a memorysuch as memory 226 that, when executed by a processor such as CPU 224,cause the computing device to perform process 700. Process 700 can alsobe implemented using hardware in whole or in part. As explained above,some computing devices may have multiple memories and multipleprocessors, and the steps of process 700 may in such cases bedistributed using different processors and memories.

For simplicity of explanation, process 700 is depicted and described asa series of steps. However, steps in accordance with this disclosure canoccur in various orders and/or concurrently. Additionally, steps inaccordance with this disclosure may occur with other steps not presentedand described herein. Furthermore, not all illustrated steps may berequired to implement a method in accordance with the disclosed subjectmatter.

At step 702, process 700 begins decoding an encoded video bitstream byidentifying a block included in a frame. At step 704, process 700 readsbits from a header associated with the block and/or bits from a headerassociated with the frame to identify the encoding parameters used toencode the block. These encoding parameters may include the encodingparameters described with respect to FIG. 6. At step 706, process 700checks to see if the block has been encoded using inter prediction. Ifso, the block is to be decoded using inter prediction, and process 700uses the encoding parameters to identify the sub-blocks of the block, ifany, in step 708, and to decode the sub-blocks using the appropriateinter prediction and inverse transform modes in step 710. The interprediction and inverse transform modes, including sizes, may be providedby the bits directly or may be discerned from other data in thebitstream. For example, the transform sizes may not be directly encodedand may be inferred from the sub-block size(s).

If inter prediction is not identified as the prediction to be used todecode the block at step 706, the block can be decoded by other decodingtechniques at step 712, for example as discussed above in relation toFIG. 5. At step 714, process 700 checks to see if processing of blocksof the frame is done. If all blocks of the frame are decoded, process700 loops back to step 702 to begin processing another block in thedefined scan order. If all blocks are decoded, process 700 ends inresponse to the query of step 714.

According to the teachings herein, the necessity of testing a blocksplit into small partitions for prediction may be reduced by performingprediction on larger partitions, and then comparing the rate distortionresulting from encoding the larger partitions using a transform of thesame dimensions versus encoding the larger partitions using a transformhaving the dimensions of the smaller partitions. If the rate distortionresulting from encoding the larger partitions using the smallertransforms shows an improvement over encoding the larger partitions withlarger transforms, motion searching may be performed on the blockresulting from the smaller partitions. Before performing the transformof the residual sub-blocks, they may be combined with other sub-blockshaving similar or the same motion information to form one or morecomposite blocks. The composite block(s) may be respectively transformedby transforms having the same dimensions to generate an encoded versionof the entire block. If the rate distortion resulting from encoding thelarger partitions using the smaller transforms does not show animprovement over encoding the larger partitions with larger transforms,motion searching may be performed on the block using partitions sizesbetween the larger partitions and the smaller partitions to providealternative encoded versions of the block. The block is encodedaccording to the smallest rate distortion cost of the available versionsof the encoded blocks.

The aspects of encoding and decoding described above illustrate someexemplary encoding and decoding techniques. However, it is to beunderstood that encoding and decoding, as those terms are used in theclaims, could mean compression, decompression, transformation, or anyother processing or change of data.

The words “example” or “exemplary” are used herein to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “example” or “exemplary” is not necessarily to be construed aspreferred or advantageous over other aspects or designs. Rather, use ofthe words “example” or “exemplary” is intended to present concepts in aconcrete fashion. As used in this application, the term “or” is intendedto mean an inclusive “or” rather than an exclusive “or”. That is, unlessspecified otherwise, or clear from context, “X includes A or B” isintended to mean any of the natural inclusive permutations. That is, ifX includes A; X includes B; or X includes both A and B, then “X includesA or B” is satisfied under any of the foregoing instances. In addition,the articles “a” and “an” as used in this application and the appendedclaims should generally be construed to mean “one or more” unlessspecified otherwise or clear from context to be directed to a singularform. Moreover, use of the term “an implementation” or “oneimplementation” throughout is not intended to mean the same embodimentor implementation unless described as such.

Implementations of transmitting station 112 and/or receiving station 130(and the algorithms, methods, instructions, etc., stored thereon and/orexecuted thereby, including by encoder 470 and decoder 500) can berealized in hardware, software, or any combination thereof. The hardwarecan include, for example, computers, intellectual property (IP) cores,application-specific integrated circuits (ASICs), programmable logicarrays, optical processors, programmable logic controllers, microcode,microcontrollers, servers, microprocessors, digital signal processors orany other suitable circuit. In the claims, the term “processor” shouldbe understood as encompassing any of the foregoing hardware, eithersingly or in combination. The terms “signal” and “data” are usedinterchangeably. Further, portions of transmitting station 112 andreceiving station 130 do not necessarily have to be implemented in thesame manner.

Further, in one aspect, for example, transmitting station 112 orreceiving station 130 can be implemented using a general purposecomputer or general purpose processor with a computer program that, whenexecuted, carries out any of the respective methods, algorithms and/orinstructions described herein. In addition or alternatively, forexample, a special purpose computer/processor can be utilized thatcontains other hardware for carrying out any of the methods, algorithms,or instructions described herein.

Transmitting station 112 and receiving station 130 can, for example, beimplemented on computers in a video conferencing system. Alternatively,transmitting station 112 can be implemented on a server and receivingstation 130 can be implemented on a device separate from the server,such as a hand-held communications device. In this instance,transmitting station 112 can encode content using an encoder 470 into anencoded video signal and transmit the encoded video signal to thecommunications device. In turn, the communications device can thendecode the encoded video signal using a decoder 500. Alternatively, thecommunications device can decode content stored locally on thecommunications device, for example, content that was not transmitted bytransmitting station 112. Other suitable transmitting station 112 andreceiving station 130 implementation schemes are available. For example,receiving station 130 can be a generally stationary personal computerrather than a portable communications device and/or a device includingan encoder 470 may also include a decoder 500.

Further, all or a portion of implementations of the present inventioncan take the form of a computer program product accessible from, forexample, a tangible computer-usable or computer-readable medium. Acomputer-usable or computer-readable medium can be any device that can,for example, tangibly contain, store, communicate, or transport theprogram for use by or in connection with any processor. The medium canbe, for example, an electronic, magnetic, optical, electromagnetic, or asemiconductor device. Other suitable mediums are also available.

The above-described embodiments, implementations and aspects have beendescribed in order to allow easy understanding of the present inventionand do not limit the present invention. On the contrary, the inventionis intended to cover various modifications and equivalent arrangementsincluded within the scope of the appended claims, which scope is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structure as is permitted under the law.

What is claimed is:
 1. A method for encoding a video stream having aplurality of frames, the method comprising: encoding a block a firsttime by encoding at least one sub-block of the block having a first sizeusing a respective inter prediction mode and a transform having thefirst size; encoding the block a second time, after encoding the blockthe first time, by encoding the at least one sub-block of the blockhaving the first size using the respective inter prediction mode and atransform having a second size smaller than the first size; performinginter prediction of the block by performing inter prediction of aplurality of sub-blocks of the block having the second size to generatea respective plurality of sub-block residuals; combining one or more ofthe plurality of sub-block residuals to form at least one compositeblock using motion information associated with respective ones of theplurality of sub-blocks; encoding the block a third time, after encodingthe block the second time, by transforming sub-block residuals of eachcomposite block of the at least one composite block using a respectivetransform corresponding to a size of each composite block; and selectingone of the encoded blocks as an output encoded block.
 2. The method ofclaim 1, wherein: encoding the block the first time generates a firstrate distortion value; encoding the block the second time generates asecond rate distortion value; encoding the block the third timegenerates a third rate distortion value; and the selecting is based onwhich of the first rate distortion value, the second rate distortionvalue, or the third rate distortion value is a lowest rate distortionvalue.
 3. The method of claim 1, wherein the block is a first block, themethod further comprising: encoding a second block a first time togenerate a first rate distortion value for the second block by encodingat least one sub-block of the second block having the first size using arespective inter prediction mode and the transform having the firstsize; encoding the second block a second time, after encoding the secondblock the first time, to generate a second rate distortion value byencoding the at least one sub-block of the second block having the firstsize using the respective inter prediction mode and the transform havingthe second size smaller than the first size; encoding the second blockat least a third time, after encoding the second block the second time,to generate a respective rate distortion value by encoding a pluralityof sub-blocks of the second block having at least a third size greaterthan the first size; and selecting one of the encoded second blocks asan output encoded block depending upon which has a lowest ratedistortion error.
 4. The method of claim 3, further comprising: encodingthe second block at least a fourth time by: generating a fourth ratedistortion value by encoding a plurality of sub-blocks of the blockhaving the third size greater than the first size; and generating afifth rate distortion value by encoding a plurality of sub-blocks of theblock having a fourth size greater than the first size.
 5. The method ofclaim 4, wherein the first size is 2N×2N, the second size is N×N, thethird size is 4N×2N and the fourth size is 2N×4N; and wherein N is aninteger greater than or equal to
 2. 6. The method of claim 1, whereinthe first size is 2N×2N, the second size is N×N and a size of eachcomposite block is one of N×N, 2N×N, N×2N, 2N×2N, N×4N, 4N×N, 2N×4N,4N×2N or 4N×4N; and wherein N is an integer greater than or equal to 2.7. The method of claim 1, wherein the output encoded block is the blockencoded the third time and the at least one composite block comprises aplurality of composite blocks having at least two different sizes, themethod further comprising: including bits within an encoded videobitstream indicating the inter prediction modes used to encode theplurality of composite blocks and the at least two different sizes. 8.The method of claim 1, wherein combining the one or more of theplurality of sub-block residuals to form the at least one compositeblock comprises: comparing the motion information associated withrespective ones of the plurality of sub-blocks; and grouping those ofthe plurality of sub-block residuals whose associated sub-blocksneighbor each other and have substantially the same motion information.9. The method of claim 8, wherein the motion information comprises amotion vector.
 10. The method of claim 8, wherein the motion informationcomprises a motion vector and a reference frame.
 11. The method of claim1, further comprising: including at least one bit within an encodedvideo bitstream indicating each inter prediction mode used to encode theoutput encoded block.
 12. The method of claim 1, wherein a size of theblock is two times the first size, the method further comprising:encoding the block a fourth time to generate a fourth rate distortionvalue by encoding the block using an inter prediction mode and atransform having the size of the block.
 13. The method of claim 1,further comprising: partitioning the block into two partitions to formthe at least one sub-block of the block having the first size beforeencoding the block the first time.
 14. A method for encoding a videostream having a plurality of frames, the method comprising: encoding ablock a first time to generate a first rate distortion value by encodingat least one sub-block of the block having a first size using arespective inter prediction mode and a transform having the first size;encoding the block a second time, after encoding the block the firsttime, to generate a second rate distortion value by encoding the atleast one sub-block of the block having the first size using therespective inter prediction mode and a transform having a second sizesmaller than the first size; responsive to the second rate distortionvalue being less than the first rate distortion value: performing interprediction of the block by performing inter prediction of a plurality ofsub-blocks of the block having the second size to generate a respectiveplurality of sub-block residuals; combining one or more of the pluralityof sub-block residuals to form at least one composite block using motioninformation associated with respective ones of the plurality ofsub-blocks; and encoding the block a third time, after encoding theblock the second time, to generate a third rate distortion value bytransforming sub-block residuals of each composite block of the at leastone composite block using a respective transform corresponding to a sizeof each composite block; and selecting an output encoded block for theblock based on the rate distortion values.
 15. The method of claim 14,further comprising: encoding the block a fourth time to generate afourth rate distortion value by encoding a plurality of sub-blocks ofthe block having a third size between the first size and the secondsize; and encoding the block a fifth time to generate a fifth ratedistortion value by encoding a plurality of sub-blocks of the blockhaving a fourth size between the first size and the second size anddifferent from the third size.
 16. An apparatus for encoding a videostream having a plurality of frames, the apparatus comprising: a memory;and a processor configured to execute instructions stored in the memoryto: encode a block a first time by encoding at least one sub-block ofthe block having a first size using a respective inter prediction modeand a transform having the first size; encode the block a second time,after encoding the block the first time, by encoding the at least onesub-block of the block having the first size using the respective interprediction mode and a transform having a second size smaller than thefirst size; perform inter prediction of the block by performing interprediction of a plurality of sub-blocks of the block having the secondsize to generate a respective plurality of sub-block residuals; combineone or more of the plurality of sub-block residuals to form at least onecomposite block using motion information associated with respective onesof the plurality of sub-blocks; encode the block a third time, afterencoding the block the second time, by transforming sub-block residualsof each composite block of the at least one composite block using arespective transform corresponding to a size of each composite block;and select one of the encoded blocks as an output encoded block.
 17. Theapparatus of claim 16, wherein a size of the block is larger than thefirst size, the instructions comprising instructions to: encode theblock a fourth time to generate a fourth rate distortion value byencoding the block using an inter prediction mode and a transform havingthe size of the block.
 18. The apparatus of claim 16, wherein theinstructions comprise instructions to: partition the block into twopartitions to form the at least one sub-block of the block having thefirst size before encoding the block the first time.
 19. The apparatusof claim 16, wherein the instructions to combine comprise instructionsto: compare the motion information associated with respective ones ofthe plurality of sub-blocks; and group those of the plurality ofsub-block residuals whose associated sub-blocks neighbour each other andhave substantially the same motion information.
 20. The apparatus ofclaim 19, wherein the motion information comprises a motion vector and areference frame and wherein sub-blocks have substantially the samemotion information when each of the sub-blocks has a common referenceframe and has a common motion vector within a defined range.